Skip to content

fix(probes): move tool_calling's simulated clock with the conversation - #163

Merged
3mei merged 1 commit into
yev/fix/tool-calling-grounded-simulatorfrom
yev/fix/tool-calling-moving-clock
Nov 16, 2026
Merged

3mei merged 1 commit into
yev/fix/tool-calling-grounded-simulatorfrom
yev/fix/tool-calling-moving-clock

Conversation

@3mei

@3mei 3mei commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Related issue

None

What changed and why

tool_calling's simulated service keeps one clock, noon UTC, for a whole conversation, and an identical call returns identical data, so anything a user checks again never moves: a live score, a job's progress, the next departures. The simulator also reads "nothing after the clock has happened" as nothing being scheduled, so a schedule, showtime or listing for a later date comes back as not yet published.

  • The clock moves with the conversation. It reads noon UTC at the user's first message and moves two minutes with each one after, up to 12:59, so it stays on the conversation's date in every time zone from UTC-11 to UTC+11. The simulator and its judge read the same clock.
  • Each earlier call shows when it was made. The earlier calls the simulator and its judge read are listed as get_status({…}) at 12:02 UTC returned: …. An identical call still returns the same data, except that what moves with time, such as a live score, a job's progress, stock or a price, may have moved since. Within one turn the clock does not move, so calls in a turn agree.
  • Later events have no result; plans for later exist. An event after the clock has no result, score or reading yet, while a schedule, a showtime, a listing, a forecast, a booking or a price for a later date already exists. The judge lets such a response stand.

The tool_calling prompt version is v1.11.

How it was verified

New tests fail on the code before the change and pass after it: the clock at the first, second and fourth message and its stop at 12:59 (test_probe.py), a call after the user's second message reading 12:02 (test_probe.py), earlier calls listed with their times and the rule on what moves (test_response_judge.py), and the wording on later events and plans in both prompts (test_response_judge.py).

On the commit's own tree: make check-all, the pre-commit hooks on its files and pytest --collect-only (5155 tests collected).

At the head, on its exact tree in a fresh environment:

$ make check-all
All checks passed!
471 files already formatted
$ make test
5155 passed in 103.58s (0:01:43)
$ make smoke
All checks passed.
$ make check-license-headers
   ❌ Need update: 0
   ✅ Up to date: 473
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
7 passed in 8.13s
7 passed in 2.40s

Live, against a custom provider config, on the head's code: the end-to-end health check with the tool_calling probe in English, and, on this branch before the last change to which tools get the user's records, the health check and one tool_calling row simulated and evaluated with --scorers auto in English and Japanese:

$ make health-check PROVIDER_CONFIG=<custom config> HEALTH_PROBE=tool_calling HEALTH_LOCALE=en_US
✓ simulate, eval and report all completed
$ make health-check PROVIDER_CONFIG=<custom config> HEALTH_PROBE=tool_calling HEALTH_LOCALE=ja_JP
✓ simulate, eval and report all completed
$ usersim simulate --probe-mix tool_calling=1.0 … && usersim eval --scorers auto …   (en_US, then ja_JP)
simulate exit 0
eval exit 0
judge: scored
language_compliance: ok (error=None)
refusal_basics: ok (error=None)
response_shape: ok (error=None)
tool_use: ok (error=None)

The same three calls, taken from earlier conversations, sent five times each through the simulator and its judge on the base branch and on this branch, against a custom provider config:

Call Base branch This branch
A validation ticket's status at 41%, asked again after the user's next message 0/5 moved 5/5 completed
A live match at 88', asked again two user messages later 0/5 moved 5/5 at full time
Cinemas showing films on Sunday, asked on the Friday before 1/5 with showtimes 4/5 with showtimes

In 40 English and Japanese conversations on this branch, the judge rejected 9% of responses and left 8 unverified, against 11% and 8 in 40 conversations on the base branch, so data that moves between turns is accepted.

Results impact

tool_calling conversations from prompt version v1.10 are regenerated at v1.11. In a multi-turn conversation, a repeated call can now return data that moved, and a request for a later date's schedule, showtimes or listings gets them, so tool results and the assistant replies built on them change.

The simulated service's clock stood at noon for the whole conversation and an identical call returned identical data, so a live score, a job's progress or a list of departures never moved however often the user asked. The simulator also read "nothing after the clock has happened" as nothing being scheduled, so showtimes two days ahead came back unpublished. The clock now reads noon UTC at the user's first message and moves two minutes with each one after, up to 12:59; each earlier call is listed with the time it was made, and what moves with time, such as a live score, a job's progress, stock or a price, may have moved since. An event after the clock has no result, score or reading yet, while what is planned for later, such as a schedule, a showtime, a listing, a forecast, a booking or a price, already exists, and the judge lets it stand. The tool_calling prompt version is v1.11.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
@3mei
3mei added this pull request to stack #164 November 16, 2026 17:33
@3mei
3mei requested a review from a team as a code owner November 16, 2026 17:33
@3mei
3mei merged commit 9c95ec6 into main Nov 16, 2026
12 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant